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Abstract 

Higher education accountability is a field characterized by complexity. Prior 
frameworks grounded in psychometrics, economics, and history fall short 
in explaining the persistence and composition of its complexity. This article 
employs organizational theory to identify the multiple conflicting approaches of 
higher education accountability and explain their persistence. The seven identified 
fields function as specialized silos, each with a unique logic and approach toward 
accountability, they are: assessment, accreditation, institutional research, 
institutional effectiveness, educational evaluation, educational measurement, and 
higher education public policy. The seven accountability silos are systematized 
into a single conceptual model using an institutional logics framework. This 
article provides an alternative to the silo-based approach and argues that future 
accountability efforts must integrate by examining the knowledge domains of 
other silos to successfully navigate the changing environment of higher education. 
The implications of an integrated accountability approach are considered for five 
topic areas: data, the professions, structure, responsibility, and transparency. 


The Seven Silos of Accountability in 
Higher Education: Systematizing Multiple 
Logics and Fields 


Higher education does not lack accountability. Rather it lacks enough of 
the proper kind, and is burdened with too much of an unproductive kind. 
(Graham, Lyman, & Trow, 1995, p. 7) 


s. 


"ince the late 20th century, colleges and universities have had to respond to 
persistent calls from multiple social sectors about the expansion of accountability in American 
higher education. The increased reporting measures are the result of multiple contextual 
factors that have influenced the system of higher education. In part, the substantial increases 
in the cost of obtaining a college education have catalyzed the American public to question 
the value of a postsecondary degree and to call for greater transparency regarding college 
outcomes (Blumentstyk, 2015; Carey 2015; Webber & Boehmer, 2008). Additionally, many 
sectors of modern society, such as government, insurance, healthcare, and banking, have been 
subject to rising levels of standards and standardization as the primary form of regulation, 
a phenomenon higher education has been unable to avoid (Brunsson & Jacobsson, 2000; 
Busch, 2011; Lampland & Star, 2009). Finally, lawmakers have increasingly emphasized that 
resource allocation be awarded based on the performance of the organization, necessitating 
that the college or university give an account of its educational output (Dougherty et al., 
CORRESPONDENCE 2016; Doyle, McLendon, & Hearn, 2010). The many drivers of accountability have resulted 

in a complex system of higher education accountability that is comprised many disparate 
fields and approaches. 
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The compounded impact of decades of expanded accountability policies and measures 
is that universities annually collect multiple types of data at multiple levels in the organization 
to satisfy multiple regulatory agencies. Administrators and researchers coordinating these 
efforts within universities have collectively organized into multiple professional fields that 
advance “best practices” within their respective areas (Banta, Lund, Black, & Oblander, 
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1996). Seven identified fields function as specialized silos, each with a unique rationality 
and approach toward matters of higher education accountability, they are: assessment, 
accreditation, institutional research, institutional effectiveness, educational evaluation, 
educational measurement, and higher education public policy. Within the literature, these 
seven disparate silos lack engagement with one another and possess conflicting definitions 
of foundational terms. Thus, an important challenge that remains is comprehending 
both the complex social context and the many disparate approaches to higher education 
accountability. In this vein, the aim of this article is to develop a conceptual model within 
which the persistently different accountability approaches may be understood. Moreover, I 
argue that future accountability efforts must integrate by examining the knowledge domains 
of other silos in order to successfully navigate the changing environment of higher education. 


I argue that future 
accountability efforts 
must integrate by 
examining the knowledge 
domains of other silos 
in order to successfully 
navigate the changing 
environment of higher 
education. 


This article purports that the differential approaches to higher education accountability 
can be systematically understood through the lens of institutional logics. Institutional logics 
is a framework from organizational theory used to understand the responses of actors— 
organizations and individuals—which operate in complex social environments (Friedland & 
Alford, 1991). The framework gives consideration to both the internal aspects of organizations, 
as well as the external forces by which they may be influenced (Thornton, 2004; Thornton & 
Ocasio, 1999). By employing an institutional logics framework to examine the literature on 
higher education accountability, I situate the seven accountability silos within the broader 
context of the market, state, and profession. To achieve this, I first identify the multiple 
accountability silos by their respective knowledge domains and membership associations. 
Then, I argue that differences between the multiple accountability silos persist as a result of 
unique responses to the broader social institutions in which they are embedded (e.g. market, 
state, and profession). Following this, I map the complex social context of higher education 
accountability and thereby systematize the disparate silos into a single conceptual model. 
Finally, I emphasize that effective accountability responses to a changing social context must 
examine multiple logics and multiple fields. 


This article makes three notable contributions to the higher education accountability 
literature. First, by mapping across the seven fields of higher education accountability, 
it extends prior research that mapped within individual fields of accountability, such as 
assessment (Ewell, 2009) and institutional research (Volkwein, 1999). Additionally, the 
mapping of the multiple accountability silos advances the discourse beyond the dominant 
focus on histories (e.g. how we arrived here) and dichotomies (e.g. research-practice, internal- 
external, summative-formative, inputs-outputs, and accountability-improvement) that have 
been traditionally used to describe the complexity of higher education accountability but 
fall short of explaining its continued persistence and broader composition (Gaston, 2014; 
Marchand & Stoner, 2012; Reichard, 2012; Suskie, 2015; Upcraft & Schuh, 2002; Zumeta & 
Kinne, 2011). Finally, employing institutional logics to systematize the field encourages the 
discourse to focus on integration by giving consideration to the sources of rationality and the 
disparate responses of the individual accountability silos to broader social institutions. 


Overview of the Field 


As new policies and regulatory agencies formed over time, universities established 
offices, practices, and routines within the organization to give an account to various external 
groups. The university personnel tasked with the oversight of different facets of organizational 
accountability gradually organized into collective membership associations that provided 
individuals the opportunity to make sense of their various practices and routines (Volkwein, 
2008). Membership associations at the state, regional, and national levels were essential in 
establishing the boundaries of a field given that they maintained and perpetuated distinct 
norms, networks, vocabularies, and practices (Reichard, 2012). Membership associations are 
an important characteristic for distinguishing the disparate fields of accountability given that 
a field can be identified by locating participants who cohere around a common purpose and 
carry out exercises that cut across organizations (Thornton, Ocasio, & Lounsbury, 2012). 
Each association maintains various types of publications in order to communicate with its 
members, maintain established norms, advance best practices, and sustain the existence of its 
knowledge domain. Furthermore, each field possesses characteristics that distinguish it from 
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the other approaches, which include: a unique discourse, scholarly or professional journals, a 
handbook of research, and a published history of its development. 

When the knowledge domain of the membership associations and its scholarly 
literature are comparatively examined, they may be used to identify the various approaches 
to higher education accountability. Upon examining these, I identified seven fields of higher 
education accountability: assessment, accreditation, institutional research, institutional 
effectiveness, educational measurement, educational evaluation, and higher education 
public policy. Within a specific college or university, these fields inform the differential 
organization of individuals, groups, teams, committees, or entire offices. The membership 
associations affiliated with each accountability field are composed of the many like-minded 
university employees who have collectively organized around a specific set of shared values, 
practices, and content (see Table 1). For example, the publications, emphases, and overall 
accountability approaches by the Association for Education Finance and Policy (AEFP), a 
policy research organization, are notably different than those of the Southern Associations 
of Colleges and Schools Commission on Colleges (SACSCOC), a regional accrediting 
organization. The grouping of the various associations highlight the differences in the seven 
approaches toward higher education accountability. 

Table 1 

Affiliated Accountability Associations 


Accountability Silo 

Affiliated Associations 

Assessment 

National Institute for Learning Outcomes Assessment (NILOA) 

Assoc, for the Assessment of Learning in Higher Education (AAHLE) 

Association of American Colleges & Universities (AAC&U) 

Accreditation* 

Regional accreditation: Southern Association of Colleges and Schools (SACS) 
Disciplinary accreditation: American Bar Association (ABA) 

Institutional Research 

Association for Institutional Research (AIR) 

Society for College and University Planning (SCUP) 

National Association of College & University Business Officers (NACUBO) 

Institutional Effectiveness 

Association for Higher Education Effectiveness (AHEE) 

Education Measurement 

National Council on Measurement in Education (NCME) 

Evaluation 

American Evaluation Association (AEA) 

Higher Education Public Policy 

Association for Education Finance and Policy (AEFP) 

Association for Public Policy Analysis & Management (APPAM) 

Council on Public Policy and Higher Education (CPPHE-ASHE) 


*For a fuller list of accreditation-related associations, please see Higher education accreditation: How it’s changing, 
why it must, by P.L. Gaston, p. 205-222. Copyright 2014 by Sterling, VA: Stylus. 


Not only do differences exist between the seven fields of higher education accountability, 
but they function as individualistic silos. Within the literature, many of the fields do not 
engage one another or give consideration to the other forms of addressing accountability in 
higher education. For example, the field of assessment engages accreditation on matters of 
student learning outcomes, but takes issue with the use of enrollments and alumni salaries as 
a meaningful outcome, one that is primarily used in the field of higher education public policy 
(Baum, May, & Payea, 2013; Gross & Berry, 2016; Schneider, 2016). Furthermore, solutions 
to ideological tensions or the improvement of practices are predominantly limited to within- 
silo perspectives. For example, the field of assessment ardently advocates that members 
strive to develop and strengthen a “culture of assessment” within individual organizations, 
rather than give consideration as to how present approaches might be integrated with other 
accountability silos (Fuller, 2013; Fuller, Skidmore, Bustamante, & Holzweiss, 2016; Ndoye 
& Parker, 2010). The entrenchment and persistence is due, in part, to individual fields of 
higher education accountability drawing their logics from the broader social institutions that 
they engage—the market, state, and profession. These same social institutions are those 
which have called for the further accountability of higher education. Therefore, to further 
understand the disparate responses of the multiple accountability silos and their persistent 
differences, one must examine the broader social institutions in which they are embedded. 


The social institutions 
of the market, state, and 
profession are those to 
which higher education 
must give an “ account” 
of its use of resources 
and achievement of 
outcomes. 
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College Scorecard 
emphasizes alumni 
salaries (e.g. market 
logic) as valid outcome 
data for a university, 
whereas the AAC&U 
VALUE rubrics 
emphasize student 
learning (e.g. 
professional logic) 
as valid outcome 
data for a university. 

Subject to both of 
these accountability 
approaches, a given 
university must navigate 
the tensions between 
the two logics. 


External Influences on the Field 

Prior accountability research has made significant advancements with regard to 
developing instruments (Shavelson, 2010), articulating histories (Ewell, 2011; Reichard, 
2012), establishing best practices (Banta, Lund, Black, & Oblander, 1996), and employing 
novel methodologies (Doyle, McLendon, & Hearn, 2010; Murnane & Willett, 2010). However, 
these approaches are unable to examine the social processes and complex social context of the 
higher education accountability sector. I propose that organizational theory provides scholars 
with the necessary conceptual tools to understand the complex environment in which various 
actors—organizations and individuals—function. 

More specifically, the institutional logics framework organizes the social embeddedness 
of actors in order to examine how they are influenced by their social context. The institutional 
logics perspective assumes that society is not of a singular logic; rather it is comprised of 
multiple broad social institutions, such as the market, state, profession, family, and religion 
(Friedland & Alford, 1991; Thornton, 2004). Moreover, each social institution possesses a 
specific logic and provides the actors embedded within them with unique ways to order their 
practices, vocabularies, values, and identities (Thornton & Ocasio, 2008). The institutional 
logics perspective permits researchers to identify the source of differences in practices, 
discourses, and identities that exist between fields by taking the broader social institutions 
into consideration. Given that the distinct logic of one social institution (e.g. market) conflicts 
with the logic of another social institution (e.g. profession), actors must consistently address 
tensions between the multiple institutions in which they are embedded. For example, College 
Scorecard emphasizes alumni salaries (e.g. market logic) as valid outcome data for a university, 
whereas the AAG&U VALUE rubrics emphasize student learning (e.g. professional logic) as 
valid outcome data for a university. Subject to both of these accountability approaches, a given 
university must navigate the tensions between the two logics. 

Researchers and practitioners of higher education accountability must identify the 
respective social institutions in which they are embedded if they are to successfully navigate 
the extant tensions between the various logics. Higher education scholars have continually 
referred to the same trio of social institutions that influence the postsecondary landscape: the 
market, state, and profession (Clark, 1983; Slaughter & Rhoades, 2004; Rhoads & Torres, 2006). 
Joseph Burke (2005) identified this triad of social institutions as those which most influenced 
the interests, pressures, and priorities of higher education accountability and deemed it the 
“accountability triangle.” The social institutions of the market, state, and profession are those 
to which higher education must give an “account” of its use of resources and achievement 
of outcomes. To further understand the role of these social institutions within the higher 
education accountability context, a typology for each social institution is briefly discussed 
below. The typologies are a compilation of the select elements and categories of each social 
institution, as found within the literature. 


Market 


The institution of the market refers to the social sphere where the exchanges of 
goods and services occur between buyers and sellers (Scott & Marshall, 2009). The norms 
of the market center on self-interest and seek to benefit individual actors, whereas its 
strategy emphasizes the efficiency of transactions (Thornton, 2004). As discussed in the 
higher education accountability literature, the root metaphor of the market logic emphasizes 
performance (see Table 2). Researchers focus on changes in “performance funding” across 
states and organizational types. “Performance targets” are monitored via data dashboards 
using “key performance indicators” (KPIs) such as transfer rates, enrollment data, cost- 
per ratios, grant funding, and research output, among many others (Massy, 2016). Broader 
“institutional performance” is comparatively examined via organizational benchmark data 
such as graduation rates, alumni salaries, and endowment performance. Market norms are 
distinct from, and often at odds with, the norms of the profession (Stone, 2002). Scholars 
have devoted significant attention to examining the impact of the market on the profession of 
higher education (Slaughter & Rhoades, 2004). The market’s expanding influence has brought 
about substantive changes in the financing of a college degree (Doyle, 2006; Doyle, McLendon, 
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Table 2 

Institutional Typologies for Market, State & Profession 


Key Characteristics 

Market 

State 

Profession 

Root metaphor 

Performance 

Compliance 

Learning 

Basis of norms* 

Self interest 

Citizenship in nation 

Membership in guild 

Basis of strategy* 

Increase efficiency of 
transactions 

Increase community good 

Increase reputation and 
quality of craft 

Organizational form* 

Marketplace 

Legal bureaucracy 

Network organization 

Data focus 

Data that illustrates 
outputs, growth, or return 
on investment 

Data that illustrates 
adherence to policies and 
standards 

Data that illustrates 
student learning or 
development 

Data treatment 

Analysis of efficiency or 
causality 

Presentation of frequency 
data or narrative argument 

Examination of pre/post 
change or 

formative/ summative 


*Denotes a categorical element adapted from Markets from culture: Institutional logics and organizational decisions 
in higher education publishing by P.H. Thornton, p.44-45. Copyright 2004 by Stanford, CA: Stanford University 
Press. 


State 


The social institution of the state refers to the collective set of agencies (e.g. armed 
forces, civil service, judiciary, etc.) that possess the authority to govern a society (Scott 
& Marshall, 2009). The norms of the state center on citizenship within one’s nation, 
while its form is organized around the concept of legal bureaucracy (Thornton, 2004). As 
discussed in the higher education accountability literature, the root metaphor of the state 
logic emphasizes compliance. The state logic focuses on the “disclosure” of information to 
highlight conformity with “regulations” and “standards” established by the government or 
their respective monitoring agencies (Brunsson & Jacobsson, 2000). To determine whether 
the compliance of standards has been achieved, federal agencies rely on two types of 
approaches. First, agencies oversee the annual collection of quantitative data such as the 
Integrated Postsecondary Education Data System (IPEDS), Internal Revenue Service (IRS) 
Form 990, and the Glery Act, to name a few. Second, the agencies coordinate the systematic 
review of legal narratives, which make the case for compliance or adherence to standards, 
such as state authorizations for operation (Ewell, Boeke, & Zis, 2010). This dual approach 
reinforces the legal bureaucratic form that organizes the social sphere as well as the fields 
and organizations drawing from its logic. 

Profession 

The social institution of the profession refers to a type of work orientation or work 
organization for a specific interest group. Entrance into or membership in the group is 
monitored by a code of conduct, practice, or values (Scott & Marshall, 2009). As discussed in 
the higher education accountability literature, the root metaphor of the professional logic of 
higher education emphasizes learning . The professional logic focuses on measuring aspects 
related to the learning of students, including the improvement of learning. Examining 
learning often takes the form of educational measurement or educational assessment. A 
measurement approach focuses on employing psychometric techniques to quantify learning, 
knowledge, or cognitive development (Shavelson, 2010), whereas an assessment approach 
focuses on examining “student learning outcomes” or “essential learning outcomes” 
(Arum, Roksa, & Cook, 2016, p. 4). Here, educators are encouraged to utilize “high impact 
educational practices” and employ “learning outcomes assessments” to examine their 
influence. Historically, the role of the profession was to establish the quality of craft and 
safeguard the quality from the influence of the market (Thornton et ah, 2012). However, 
scholars have noted that contemporary professions (education, architecture, accounting, 
etc.) increasingly must confront the expanding influence of the market logic into the 


The term [institutional 
effectiveness] addresses 
the systematic examina¬ 
tion of the planning 
and decision making in 
multiple areas across 
the university (adminis¬ 
trative, educational, etc.) 
and at multiple levels 
in order to determine 
its effectiveness as an 
organization. 
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professional domain (Hermanowicz, 2011; Thornton, Jones, & Kury 2005). Therefore, future 
higher education accountability efforts that use measures of learning must acknowledge 
their limited application when responding to an expanding market logic. 

The influence of the social institutions of the market, state, and profession has 
garnered the attention of scholars of higher education (Clark, 1983; Slaughter & Rhoades, 
2004) and more specifically, higher education accountability (Burke, 2005). This triad 
of social institutions is the source of logics, order, and rationality for the accountability 
fields and universities embedded within them. Because the embeddedness of fields and 
organizations is not mutually exclusive, they are often influenced by multiple logics. 
Therefore, understanding the differential approaches of higher education accountability is 
dependent upon identifying the dominant logic or combinations of logics upon which or to 
which they primarily respond. 

Mapping Multiple Logics and Fields 

Each of the seven accountability silos is a distinct field supported by a vast literature 
and a large number of scholars and practitioners in membership associations. Given that 
limited engagement occurs across the disparate silos, higher education possesses a complex 
system of accountability that warrants further clarity. This section seeks to converge the 
seven identified silos and the broader social institutions into a single conceptual model. The 
differential sources of rationality that ultimately influence the varied approaches to higher 
education accountability are highlighted by mapping the complex social context. 

Assessment 


As a field, institutional 
effectiveness can be 
described as combining 
the processes of 
accreditation (state 
and profession) with 
an added emphasis 
on organizational 
performance (market). 


Assessment is the systematic collection, analysis, and translation of evidence 
on a given topic or outcome (Astin & antonio, 2012; Seclosky & Denison, 2012; Suskie, 
2004). Assessment gives priority to student learning, whether the process is led by 
administrators or faculty, or focused on curricular or co-curricular characteristics of the 
university. Scholars have specifically noted that assessment is distinct from the other higher 
education accountability fields of accreditation, measurement, and evaluation (Gaston, 
2014; Seclosky & Denison, 2012; Suskie, 2015). The norms of the profession center on 
collectively established codes such as “Principles of Good Practice for Assessing Student 
Learning” or “Seven Principles for Good Practice in Undergraduate Education” (Banta, 
Lund, Black, & Oblander, 1996; Ghickering & Gamson, 1987; Upcraft & Schuh, 1996). The 
field of assessment is predominantly comprised of administrators tasked with the oversight 
of measuring student learning outcomes across the curriculum of academic disciplines 
within individual colleges and universities. Recent works encourage faculty to become more 
involved in the processes of establishing student learning outcomes (Arum, Roksa, & Cook, 
2016). Efforts also emphasize the assessment of student learning outcomes across student 
affairs and co-curricular areas of the university (Bresciani, Gardner, & Hickmont, 2010). 

The field of assessment draws its distinct rationality from the professional logic 
whose root metaphor emphasizes learning (see Figure 1). The broader social institution of 
the profession guides the responses the field of assessment adopts toward higher education 
accountability. Data are collected on individual assignments and examined through rubrics 
in order to determine the extent of student learning. Some choose to showcase student 
learning through e-portfolios to highlight the array of development or competencies across 
multiple knowledge domains. Others employ the use of course-embedded techniques in order 
to strengthen the authenticity of the results. Furthermore, the aim of assessment is often 
stated to be for purposes of improving teaching and learning, as well as for accountability 
(Ewell, 2009). The applied and real-world nature of the collected data usually limits its 
generalizability beyond the context of the specific university. These organized responses 
by the field of assessment address questions of accountability that are of interest to the 
profession in an applied manner, and do not address accountability paradigms of interest to 
the state (e.g. compliance) or market (e.g. performance), as will be shown with some of the 
remaining silos. 
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Institutional Logics 


Accountability Fields 



Figure 1 . A Map of the Social Embeddedness of Higher Education Accountability 

Accreditation 

A persistent challenge 
for the field of higher 
education public policy 
has been to connect 
market-focused and 
state-focused variables 
with learning-focused 
variables more robust 
than graduation rates. 


The field of accreditation draws its distinct rationality from a combination of 
two logics—the professional logic (learning) and the state logic (compliance). The dual 
combination of these logics informs the response the field of accreditation maintains toward 
higher education accountability. The self-regulatory processes of accreditation originated as 
a form of professional accountability upheld by the ethic of peer review (Gaston, 2014). In 
constructing arguments for accrediting bodies, universities must provide evidence to argue 
their compliance with the standards of the accrediting bodies that function as a stand-in for 
the federal government (Ewell, 2011). The process resembles the practice of peer-review 
journals whereby the publication of knowledge is governed via the review of evidence by one’s 
peers within the profession. However, declines in public trust across many professions in 
society have also negatively impacted the perceived legitimacy of the self-regulatory processes 
of accreditation (Blumenstyk, 2015; Ewell, 2008). Over time, the process has evolved into an 
organized “federal regulation of academic practices” (Newell, 2012, p. 36). 


Accreditation is the peer review process established to examine the educational 
quality of colleges and universities and to ensure their adherence to standards of practice 
(Bogue & Hall, 2003; Eaton, 2012; Gaston, 2014). More formally it is, “a process by which an 
institution of postsecondary education evaluates its educational activities, in whole or in part, 
and seeks an independent judgement to confirm that it is substantially achieving its objectives 
and is generally equal in quality to comparable institutions of postsecondary education” 
(Young, Chambers, & Kells, 1983, p. 21). Three types of accreditors comprise the U.S. system 
of accreditation: regional accreditors accredit entire colleges, specialized accreditors accredit 
specific academic programs, and national accreditors accredit entire colleges that are not 
eligible for regional accreditation (Suskie, 2015). While there are many accrediting agencies 
in each of the three types, they are all “owned and operated” by the colleges and universities 
which comprise their membership (Suskie, 2015). 
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Institutional Research 


Given that the field 
of accreditation is 
organized around the 
professional system of 
peer review, it will face 
increased legitimacy 
challenges should the 
societal distrust and 
decline continue. 


The field of institutional research is comprised of persons and groups whose 
function within universities is to conduct research in order to “provide information which 
supports institutional planning, policy formation, and decision making” (Saupe, 1990, 
p. 1). The “typical” functions of institutional research address four areas within colleges 
and universities: (a) external and internal reporting, (b) planning and special projects, (c) 
data management and technical support, and (d) research and development (Volkwein, 
Liu, & Woodell, 2012). Data to support these four areas are predominantly queried from 
existing sources of information within the organization. For example, most data reporting 
requirements for the Integrated Postsecondary Educational Data System (IPEDS) or U.S. 
News & World Report rankings are fulfilled by institutional research offices or personnel. 

The field of institutional research draws its distinct rationality from a combination of two 
logics—the state logic (compliance) and the market logic (performance). The two logics influence 
the response the field of institutional research has toward higher education accountability, one 
that has been characterized as “organizational intelligence” (Terenzini, 1993). The various 
practices of organizational intelligence aim to provide service and support to faculty members, 
administrators, and coordinating groups (Stecklein, 1971). Recent advancements in technology 
and data analysis have enabled institutional researchers to further support university planning 
and decision making through the use of data mining, predictive analytics, business analytics, 
and data dashboards (McLaughlin, Howard, & Jones-White, 2012). These approaches permit 
institutional researchers to examine organizational data in innovative ways that offer the 
potential of cross-silo efforts toward higher education accountability. 


Institutional Effectiveness 


Institutional effectiveness is a “multifaceted construct with a myriad of meanings and 
interpretations” (Alfred, 2011, p. 104). Of the seven silos, it is the most widely misunderstood 
within the literature given that the term describes a university process, office, and field and 
is frequently conflated with assessment (Head, 2012). Institutional effectiveness originated 
in the mid-1980s when the Southern Association of Colleges and Schools Commission on 
Colleges (SACSCOC) regional accrediting body implemented the term to describe a new 
emphasis within its stated policies (Ewell, 2012). The notion of assessment had become 
too contentious and policy makers wanted a “broader and more acceptable” term (Rogers, 
1997). Thus, the notion was distinctively different from assessment given that “institutional 
effectiveness” was to examine all aspects of the university, whereas the field of assessment 
had limited itself to examining student learning outcomes and development. The term 
addresses the systematic examination of the planning and decision making in multiple areas 
across the university (administrative, educational, etc.) and at multiple levels in order to 
determine its effectiveness as an organization. While different from assessment, the new 
term strengthened the relationship between assessment and accreditation. By the end of the 
1980s, each of the six major accrediting bodies had adopted similar language and practices 
to evaluate the institutional effectiveness of colleges and universities (Ewell, 2011). 

The field of institutional effectiveness draws its unique rationality from all three 
institutional logics—the professional logic (learning), state logic (compliance), and market 
logic (performance). Of the seven accountability silos, it is the only one to draw from all 
three social institutions to inform its response toward higher education accountability. 
What differentiates institutional effectiveness from the silos of accreditation or institutional 
research is that it draws its rationality from each of the three social institutions, whereas 
they draw from two. As a field, institutional effectiveness can be described as combining the 
processes of accreditation (state and profession) with an added emphasis on organizational 
performance (market). Although it may be one of the more nebulous silos, institutional 
effectiveness is identified herein as a field unto itself because over time its meaning has 
expanded from a policy in a single agency to one that possesses a membership association, 
professional journal, and published history of its development. 
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Educational Measurement 


Measurement focuses on employing psychometric theories and principles to collect 
student-level educational data using test instruments. The gathering of data based on test items 
or judgements from examinees enables researchers to make decisions based on inferences 
from the data (Secolsky & Denison, 2012). Although college entrance exams, such as the ACT 
or SAT instruments, have received significant scrutiny as of late, they are the most common 
test instrument used in higher education (Lemann, 2000; Soares, 2011). Other areas of the 
university that frequently employ the testing approaches of educational measurement are skills 
assessments for incoming students, major field tests, and career aptitude tests. The majority of 
these tests are standardized instruments that have been created using an array of psychometric 
properties that include: classical test theory, item response theory, generalizability theory, 
reliability, validity, scaling, norming, and statistical modeling to name a few. In Shavelson’s 
(2010) four eras of learning assessment, the origins of standardized testing in the American 
university (1900-1933) precede the era of external accountability (1979-present). Therefore, 
it is important to note that the uses of educational measurement and testing have changed 
over time with the changing social context of higher education. 

The field of educational measurement draws its distinct rationality from the 
professional logic, whose root metaphor emphasizes learning. The broader social institution 
of the profession guides the response that the field of educational measurement adopts 
toward higher education accountability. While assessment and education measurement 
both share the singular professional logic with a focus on learning, the two are distinct 
fields with different approaches to accountability. Many of the instruments used in the 
field of education measurement are external learning assessments (e.g. standardized). 
The instruments are predominantly indirect measures given to students at a single point 
in time (e.g. SAT, ACT, MFAT, etc.), although some standardized instruments (e.g. College 
Learning Assessment) have been issued at two points in time or longitudinally in order to 
address change in student performance (Shavelson, 2010). On the other hand, the field of 
assessment tends to emphasize internal learning assessments—direct measures to examine 
the improvement of student learning outcomes. 


Educational Evaluation 

The Evaluation Thesaurus defines evaluation as, “the process of determining 
the merit, worth, or value of something, or the product of that process.” (Scriven, 1991). 
Educational evaluation addresses the fidelity of educational programs, or more specifically, 
asks whether what was proposed and what was delivered were in alignment. Focal data for 
evaluation exist at the program level, whereas the focal data for the field of educational 
measurement resides at the student level. Grayson (2012) identifies three types of 
educational evaluation: formative, summative, and developmental. Formative evaluation 
focuses on improvement through constructive feedback to program implementers and 
clients. Summative evaluation emphasizes the measurement of inputs and outcomes in order 
to determine performance and impact. Developmental evaluation focuses on identifying the 
effects of innovation and testing hypotheses in situations of uncertainty and complexity 
(Patton, 2011). An important difference between evaluation and the field of assessment is 
that evaluation uses information to make an informed judgement regarding performance and 
efficiency (Suskie, 2004). Process measures provide data that address program quantity (e.g. 
how much did we do?) and outcome measures provide data that address program impact 
(e.g. how successfully did we do it?) (Friedman, 2007). 

The field of educational evaluation draws its distinct rationality from a combination 
of two logics—the professional logic (learning) and the market logic (performance). The 
two logics influence the response the field of educational evaluation has toward higher 
education accountability. Efficiency is a fundamental aspect of educational evaluation that 
focuses on the examination of the various costs and resources (e.g. money, facilities, people, 
etc.) a given program expends in relation to its value or benefits (Boulmetis & Dutwin, 
2005). In addition to examining the efficiency of a program, evaluators also address its 
impact on participants and “return on investment” for the organization (Kirkpatrick, 1994). 


While accrediting 
bodies and individual 
universities may object 
to these market- and 
state-oriented approach¬ 
es to accountability, the 
societal distrust in the 
professions creates a 
dilemma with regard 
to maintaining past 
practices predominantly 
informed by the profes¬ 
sional logic of learning. 


Volume Eleven I Anniversary Issue 2017 


•RPA 


49 



RESEARCH & PRACTICE IN ASSESSMENT 


The emphasis of evaluators on the relationship between resources and program outcomes 
highlights the two logics that influence the field—the market logic and professional logic. 


Higher Education Public Policy 

The field of higher education public policy examines the relationship between 
resources and education-related policies. Researchers are interested in the ways in which 
resources are both generated and distributed across systems and how these processes 
relate in various ways to local, state, and federal policies on education. The field of higher 
education public policy excels at producing timely research on specific education policies 
such as performance-based funding (Bouge & Johnson, 2010; Hillman, Tandberg, & Gross, 
2014) and federal financial aid (Gross & Berry, 2016). The field also examines the extent to 
which specific educational policies migrate or diffuse across districts, organizations, states, 
and regions over time (Dougherty, Natow, Bork, Jones, &Vega, 2013; Doyle, 2006; Doyle, 
McLendon, & Hearn, 2010). Focal data for the field of higher education public policy are 
rarely individual case studies of organizations; rather, employed data are more commonly 
large scale data sets that enable researchers to examine the broader impact of education 
policies and accountability phenomena. 

The field of higher education public policy draws its distinct rationality from a 
combination of two logics—the market logic (performance) and the state logic (compliance). 
The two logics influence the response the field of higher education public policy has 
toward higher education accountability. The dominant paradigm of the field is supported 
by frameworks from economics and public policy, whereas the dominant paradigm of the 
field of educational measurement is supported by psychometric theories. Consequently, the 
variables of interest to researchers in this field also differ from other accountability silos 
as they examine alumni earnings data, graduation rates, financial aid, resource allocation, 
resource acquisition, and transfer rates among many others. A persistent challenge for the 
field of higher education public policy has been to connect market-focused and state-focused 
variables with learning-focused variables more robust than graduation rates. 


Synthesis & Evaluation of the Fields 


Applying an integrated 
paradigm may provide 
insights as to how fields 
may respond proactively, 
rather than reactively, 
to market and state 
pressures to govern 
the profession. 


The review of the higher education accountability literature across each of the seven 
fields highlights three notable characteristics: ambiguity of terms, engagement with other 
fields, and absent linkages. First, within the literature, scholars repeatedly acknowledged 
difficulty in consistent definitions of germane terms such as assessment, accountability, and 
institutional effectiveness (Alfred, 2011; Head, 2012; Wall, Hursch, & Rodgers, 2016; Zumeta, 
2011). At times, definitions explicitly conflicted with one another: assessment is comprised 
of measurement and evaluation (Astin & antonio, 2012) versus assessment is distinct from 
measurement and evaluation (Suskie, 2004). Some noted that the rapidly changing context 
of higher education impacts the identity and vocabulary of the field because it is continually 
responding to new and changing conditions within the regulatory and resource environments 
(Krist, Jones, & Thompson, 2012). Arguably, the persistent confusion is not necessarily due 
to a dynamic or changing social context but rather a lack of understanding of the social 
composition of the context and its influence upon those fields and organizations embedded 
within it. Mapping the field addresses the ambiguity of terms by establishing typologies 
for the entire sector—both the social institutions and the individual accountability silos. 
Furthermore, mapping the field shifts the discourse to focus on sources of rationality and 
the disparate responses of the individual fields of higher education accountability. 


A second notable characteristic within the literature addresses the degree of 
engagement one field has with that of another. In rare instances, the knowledge domains 
of fields engage with one another, whereas in most instances there was little engagement 
at all. In reviewing these occurrences, interaction between silos can be explained based 
on commonalities or differences in logics. The limited number of fields that engage one 
another are those which share a root logic or combination of root logics: (a) assessment 
and education measurement (profession) and (b) institutional research and higher 
education public policy (state and market). These silos share institutional logics and 
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thus share a common understanding, vocabulary, and rationality about the context of 
higher education accountability. 

A final notable characteristic pertains to the lack of linkages between social 
institutions and specific accountability silos. As discussed above, the knowledge domains of 
some silos possessed common root logics, or “linkages.” However, some accountability silos 
do not share common linkages between social institutions. In part, these missing linkages 
explain the extant tensions between the accountability silos. The field of accreditation, 
which is missing a link to the market logic, has recently been challenged with alternative 
methods of accountability that emphasize performance, return on investment, and alumni 
salaries (Fain, 2016; Stratford, 2015). The field of assessment, which is missing links to 
the state and market, must continually defend itself as being a legitimate form of reviewing 
higher education outcomes. Higher education public policy, which is missing the link to the 
profession, is efficient with its outcomes but has been unable to provide evidence of learning 
in ways that appease assessment professionals. A few select scholars have attempted to 
address topics related to absent linkages (Wall et al., 2014) but further work is necessary to 
integrate the disparate accountability approaches. 

Discussion & Future Implications 

The present state of higher education has been one characterized by persistent 
change. While prior works have discussed how individual silos of accountability have evolved 
over time as a result of the changing context, the works have not given consideration as to how 
the broader changes might impact multiple silos (Gaston, 2014; Ewell, 2008; Suskie, 2015; 
Volkwein, 2008). This final section provides an alternative to the silo-based emphasis toward 
higher education accountability. It advocates that future efforts must integrate by engaging 
the knowledge domains of other silos in order to successfully navigate the changing social 
context. It addresses five important topics: data, the professions, structure, responsibility, 
and transparency. 

Data is an important topic that pertains to each of the seven silos. In this vein, it 
is worth considering how the various fields of higher education accountability respond as 
new data types emerge. Recent education research has started to engage the development of 
analytics, more commonly referred to as “big data.” The scope of this type of data addresses 
learning analytics (Baker & Corbett, 2014), business analytics, predictive analytics (Denley, 
2014), action analytics (Milliron, Malcolm, & Kil, 2014), and social analytics (De Laat & 
Prinsen, 2014), among others. Traditional approaches to assessment have focused on the use 
of rubrics and e-portfolios as a form of examining the extent of student learning. In contrast, 
learning analytics provide researchers with new forms of learner data that are both broad 
and deep, thereby potentially enriching assessment processes (Thille et al., 2014). Applying 
an integrated paradigm to the data may make it possible to connect learning analytics (logic 
of learning) with business or predictive analytics (logic of performance) in innovative ways. 
Furthermore, the broad and deep nature of analytics introduces possibilities to aggregate 
and disaggregate data in ways that may benefit multiple silos with a single type of data. 

A second notable topic to consider is the increasing societal distrust and decline 
of the professions (Ewell, 2008; Thornton, Jones, & Kury 2005). Given that the field of 
accreditation is organized around the professional system of peer review, it will face increased 
legitimacy challenges should the societal distrust and decline continue. More recently, 
alternatives to hold universities accountable have focused on market- and state-oriented 
solutions. The required annual reporting of annual alumni salary data to gauge “return on 
investment” asserts a market based form of accountability. In contrast, the establishment 
of the United States Education Dashboard by the Department of Education to generate 
comparative organizational metrics asserts a state-based form of accountability. While 
accrediting bodies and individual universities may object to these market- and state-oriented 
approaches to accountability, the societal distrust in the professions creates a dilemma with 
regard to maintaining past practices predominantly informed by the professional logic of 
learning. Applying an integrated paradigm may provide insights as to how fields may respond 
proactively, rather than reactively, to market and state pressures to govern the profession. 


In an effort to further 
reduce costs, attempts 
to structure the system 
of higher education 
accountability should 
occur beyond individual 
universities and give 
consideration to 
redundancies across 
silos. As we aggregate 
data, is it also 
possible to aggregate 
accountability systems? 
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Whether an organization 
chooses a centralized, 
decentralized, or other 
approach to assigning 
responsibility for 
accountability mandates, 
senior administrators 
must also give 
consideration to how 
the broader seven silos 
will influence university 
decisions, as well as 
the changing context of 
higher education. 


Third, given the multiple approaches to higher education accountability and 
limited financial resources for universities, the structure of offices and personnel fulfilling 
accountability requirements within a university should also be considered. Individual 
universities should consider exploring areas of redundancy and duplication of labor with 
regard to fulfilling accountability mandates. The alignment of offices, committees, employees, 
reports and data queries could assist with reducing “administrative bloat” and associated 
organizational costs (Blumenstyk, 2015; Kirk, 2014). In a recent study, an elite research 
university calculated that its cost of complying with accountability mandates totaled $146 
million annually, or approximately $11,000 per student (Woodhouse, 2015). Many policy 
makers vehemently criticized the study, but the university asserted that its broader purpose 
in conducting the research was to emphasize that matters of compliance costs and efficiencies 
have seldom been part of the national accountability discourse (Moran, 2015). In an effort 
to further reduce costs, attempts to structure the system of higher education accountability 
should occur beyond individual universities and give consideration to redundancies across 
silos. As we aggregate data, is it also possible to aggregate accountability systems? These 
broader initiatives to identify redundancy within and across fields could begin to identify 
commonalities in structures, division of labor, and cost containment. 

Similarly, a fourth important topic to consider is that of responsibility. Specifically, 
with such disparate accountability approaches within the university, who represents the 
organization with regard to accountability? The notion of accountability and its respective 
practices impact a variety of employees within individual colleges and universities. A 
centralized approach might appoint a senior administrator to oversee all accountability efforts 
for the university in a similar manner that a chief information officer (CIO) represents the 
university on matters of information technology or a chief financial officer (CFO) represents 
the university on matters of finance and budgeting. In contrast, a decentralized approach might 
distribute the responsibility to fulfill accountability mandates to the respective university 
offices such that the assessment office addresses student learning outcomes, institutional 
research fulfills data mandates for the Integrated Postsecondary Educational Data System 
(IPEDS), campus safety addresses Glery Act compliance, and an associate provost handles 
matters pertaining to regional accreditation, to name a few. Whether an organization chooses 
a centralized, decentralized, or other approach to assigning responsibility for accountability 
mandates, senior administrators must also give consideration to how the broader seven silos 
will influence university decisions, as well as the changing context of higher education. 

A final topic of importance is that of transparency. A significant portion of the 
public scrutiny regarding higher education, and the field of accreditation more specifically, 
pertains to its “black box” nature (Gaston, 2014). From the public vantage point, there is a 
degree of uncertainty about exactly what happens inside a college or university regarding its 
resource allocation, decision making, bureaucratic procedures, and value-added processes, 
among other operations. Two immediate areas of increased transparency should address 
financial data and processes. In a market context, protecting one’s financial data or delaying 
its release provides the opportunity to maintain a sustained competitive advantage over 
other organizations. With many universities competing for financial resources, it benefits 
a college or university to conceal its data, particularly its financial data. Present federal 
accountability processes for non-profit universities conceal their financial data for nearly 
three years before being made publicly available (IRS Form 990). In contrast, present federal 
accountability processes for for-profit universities require them to make the same financial 
data publicly available on a quarterly basis (Form 10K). Given that both postsecondary 
organizational types are funded using public subsidies from federal student financial aid 
(FAFSA), the transparency time delay for the use of taxpayer monies should be significantly 
reduced. Equalizing the transparency requirements for these two organizational types will 
help higher education researchers more effectively examine the behavior of universities in 
a market context before the data are obsolete. It will also provide a skeptical public with 
relevant and timely information about how universities utilize the public resources with 
which they are entrusted. 
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has recently been scrutinized for maintaining opaque peer review processes as its primary 
approach to certifying individual colleges and universities (Blumenstyk, 2015; Suskie, 
2015). In order to maintain its established legitimacy, the field of accreditation must give 
consideration as to how it will address the logics of the three broader social institutions 
(market, state, profession) in a transparent manner. In a comparative case, peer review 
journals—who share the same professional “peer review” value as the field of accreditation— 
responded to similar scrutiny by refining internal processes and establishing metrics to make 
such processes more transparent and understood by their host associations and members 
(ASA, 2016). What metrics can be made available about accrediting agencies that would 
assuage a skeptical public? Identifying relevant metrics and making those transparent may 
serve as an initial first step for accrediting agencies or other accountability silos attempting 
to address the changing context of higher education. 

Conclusion 


Twenty years ago Graham, Lyman, and Trow (1995) argued that the American 
system of higher education was not short in approaches to accountability. Rather, they 
claimed that higher education “lacks enough of the proper kind, and is burdened with too 
much of an unproductive kind” (p. 7). The multiple approaches, or silos, of higher education 
accountability have persisted for decades, resulting in increased administrative costs, 
compounded policies, and redundant practices (Blumenstyk, 2015; Kirk, 2014; Moran, 2015). 
Prior research grounded in psychometrics, economics, and history has attempted to explain 
the complexity of higher education accountability (Gaston, 2014; Marchand & Stoner, 2012; 
Reichard, 2012; Zumeta & Kinne, 2011). However, these frameworks have been unable 
to explain both the existence of multiple approaches, as well their continued persistence. 
Rather, I argue that we could more fully understand both the complexity of the environment 
and the processes of higher education accountability by employing organizational theory. 
Using an institutional logics framework, the seven accountability silos are systematized into 
a single conceptual model, which provides actors with a new paradigm for transforming 
the sector—one that suggests that strategies of change must examine multiple fields and 
multiple logics. 


The model presented in this article systematizes the complexity by identifying the 
multiple silos of higher education accountability and their broader forces. It offers a new 
paradigm, and therefore new possibilities, for thinking about the future transformation 
of the sector. Informed by the systematized model, future changes to the sector of higher 
education accountability should adopt a three-fold focus on engagement, consolidation, 
and elimination. First, engagement across multiple accountability silos must occur through 
means such as scholarly discourse, practitioner interaction, and agency coordination, among 
others. Engagement across multiple silos will be a challenge given that the established norms, 
values, and cultures of individual silos have existed for decades. Solutions to ideological 
tensions must move past the within-silo paradigms and consider the accountability 
discourses occurring in other fields if the sector is to begin to identify a “proper kind” of 
accountability. This article provided examples of how the silo of assessment might advance 
beyond strengthening organizational “cultures of assessment” or how the silo of higher 
education public policy might consider improved measures of learning beyond graduation 
or GPA. Scholars across multiple fields might take the lead in transforming the sector by 
establishing strategic collaborations that yield joint professional meetings, policy reports, 
special journal issues, or “new directions” volumes of research. Without a commitment to 
initiate engagement and discourse across the multiple silos the transformation of the sector 
will be severely limited, if not impossible. 


The model presented in 
this article systematizes 
the complexity by 
identifying the multiple 
silos of higher education 
accountability and 
their broader forces. It 
offers a new paradigm, 
and therefore new 
possibilities, for think¬ 
ing about the future 
transformation of 
the sector. 


While engagement addresses the discourse between actors in different silos, 
consolidation addresses the unification of content between different silos. Efforts to 
integrate the processes, policies, and practices of disparate accountability silos will 
ultimately highlight redundancies in the system. Policymakers and practitioners must give 
consideration as to how data predominantly used in one field may also be used to further 
inform questions of accountability in other fields. As greater numbers of postsecondary 
organizations engage in market-based practices to ensure organizational sustainability, 
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What remains crucial 
is the principle this 
article explains—deriving 
responses to address the 
changing social context 
of higher education must 
examine multiple logics 
and multiple fields. The 
silo of accreditation will 
be unable to maintain its 
legitimacy within 
society if it cannot 
sufficiently engage the 
logics of the market, 
state, and profession. 


the accountability mechanisms must correspondingly change to sufficiently examine such 
practices. Present accrediting processes, which are established on peer review norms, must 
further incorporate market-oriented approaches of evaluation. For example, accreditation 
teams could be provided with standardized IPEDS data dashboards that highlight changes 
in key variables since the previous on-site review ten years prior. Consolidating practices in 
data collection or reporting would further reduce administrative costs at the organizational 
level. However, given the scope of consolidation that is necessary, redundancies must also 
be examined at multiple levels beyond individual colleges and universities (e.g. federal, 
regional, and state levels). 

Finally, efforts to integrate the accountability silos need to call attention to 
characteristics of the sector that warrant elimination, particularly the “unproductive 
kind” which continue to burden colleges and universities (Graham, Lyman & Trow, 1995). 
Consolidation focuses on retaining the effective attributes of the accountability system 
that must remain, whereas elimination focuses on removing the ineffective attributes of 
the system. Efforts in elimination must predominantly occur at federal, state and regional 
levels, where many accountability agencies reside. For example, policy makers and 
researchers should be permitted to examine “accreditation effectiveness” in similar ways 
that the regional accrediting agencies monitor “institutional effectiveness” of organizations. 
A group of researchers representing multiple fields (e.g. assessment, evaluation, and higher 
education public policy) will examine the silo of accreditation in novel ways, particularly 
if tasked with examining strategies of elimination. These strategies—engagement, 
consolidation, and elimination—are necessary to transform an accountability system 
characterized by complexity. 

Addressing the future of higher education accountability amidst a changing societal 
context is not limited to the topics (e.g. data, the professions, structure, responsibility, and 
transparency) and examples discussed herein. A limitless number of matters will surface, 
many of which cannot be foreseen. What remains crucial is the principle this article 
explains—deriving responses to address the changing social context of higher education 
must examine multiple logics and multiple fields. The silo of accreditation will be unable 
to maintain its legitimacy within society if it cannot sufficiently engage the logics of the 
market, state, and profession. Similarly, the silo of assessment will make few advancements 
in its practice or methodology if it cannot sufficiently engage across other silos to examine 
the practices of those fields. Is it possible to advance beyond the “best practices” of an 
individual field to “best practices” of accountability that promote integration? By looking 
at multiple logics and the multiple practices across fields, scholars and practitioners can 
address gaps in ideology, apply novel methodologies, improve efficiencies, and establish 
innovative approaches in a rapidly changing context. 
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